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1.  Introduction 

As  pointed  out  by  Torre  and  Poggio  (1984)  many  problems  of  early  vision  are  ill-posed: 
unique  stable  solutions  can  be  recovered  by  several  regularization  techniques,  in  particular 
by  standard  regularization  due  mainly  to  Tikhonov  (1943,  1963).  Providing  that  solutions 
belong  to  suitable  compact  sets,  these  techniques  can  be  successfully  applied  to  a  broad 
class  of  problems  (for  a  brief  review  see  Poggio,  Torre  and  Koch,  1985),  such  as  surface 
interpolation  (Grimson  19S1,  1982;  Terzoupulos  1984),  computation  of  visual  motion  (Horn 
and  Shunck  1981,  Hildreth  1984),  recovering  shape  from  shading  (Ikeuchi  and  Horn  1981). 
lightness  (Horn  1974)  and  edge  detection  (Torre  and  Poggio  1986). 


According  to  standard  regularization  theory,  stable  solutions  can  be  recovered  quite 
simply  if  they  belong  to  a  compact  set.  Otherwise  standard  regularization  techniques  have 
to  be  applied.  These  methods  search  for  a  solution  as  close  as  possible  to  the  data  and 
belonging  to  a  compact  set  defined  by  a  suitable  stabilizing  functional.  In  both  cases,  as  we 
will  see  in  detail,  the  concept  of  compact  set  plays  a  key  role.  Very  often,  however,  some 
additional  constraints  on  the  shape  of  the  possible  solutions  are  available:  for  example  the 
solutions  may  belong  to  the  set  of  positive  functions,  as  in  the*  case  of  lightness,  or  may  be 
bounded  by  the  values  of  some  known  functions  or  may  be  piece-wise  continuous  or  piece- 
wise  constant  as  in  some  instances  of  surface  interpolation.  More  generally  it  can  be  said 
that  these  constraints  define  a  certain  subset  in  a  suitable  functional  space.  Rutman  and 
Cabral  (1974)  have  shown  that  combining  regularization  techniques  and  shape  contraints 
improves  the  correctness  of  the  numerical  solution  in  linear  integral  problems. 


In  this  note,  after  a  brief  review  of  ill-posedness  in  functional  spaces  and  in  ft",  we  show 
which  of  these  constraints  can  be  embedded  in  the  classic  regularization  theory,  and  how. 
Two  cases  are  considered  in  detail.  In  the  first  one,  shape  constraints,  forcing  the  solution 
to  belong  to  a  compact  set.  allow  a  straightforward  regularization  of  the  problem.  In  the 
'''  uid.  more  general  case  shape  constraints  define  closed  sets  that  can  be  incorporated  into 


the  framework  of  classical  regularization  theory,  where  an  appropriate  stabilizing  functional 


constrains  the  solution  to  a  compact  set,  providing  a  simple  way  in  which  some  u  jinun 
knowledge  can  be  taken  into  account.  Some  functional  subsets  corresponding  to  interest iny. 

i 

shape  constraints  are  considered. 

We  also  answer  questions  arising  in  the  numerical  solutions  of  regularized  problem". 
Since  regularization  with  shape  constraints  is  a  problem  of  constrained  minimization,  wv 
discuss  in  some  detail  the  relationship  with  mathematical  programming. 

Our  main  conclusion  is  that  shape  constraints  can  be  applied  in  regularization  theory 
provided  they  define  compact  or  at  least  closed  subsets.  The  constraints  involving  discout  i 
unities  do  not  fit  into  this  schema  while,  for  example,  monotonicity,  convexity  and  positivity 
constraints  do. 


2.  Overview:  ill-posed  problems  in  infinite  and  finite  dimensional 

spaces 


In  this  section  we  review  briefly  the  main  problems  involved  in  the  ill-posedness  of  equations 
in  infinite  and  finite  dimensional  spaces.  We  introduce  the  concepts  of  normal  solution  and 
quasi-solution  and  show  the  connection  with  uniqueness  and  existence  of  the  solution  to  a 
given  problem.  Relationships  between  ill-conditioned  and  ill-posed  problems  in  the  discrete 
are  also  examined. 

J.l.  Ill-posed  problems  in  Hilbert  spaces 


b'  t  m  consider  the  problem  of  solving  the  (’((nation 


for  j-,  where  .r  and  y  belong  to  X  and  V',  Hilbert  spaces.  The  operator  A.  defined  on 
D(.4)  C  X,  maps  D(A)  onto  R(A)  C  5U  In  many  applications  it  is  required  that  the  the 
solution  to  (2.1.1)  t)  exists,  ti)  is  unique  and  iii)  depends  continuously  on  //.  A  problem, 
whose  solution  satisfies  ?  ),  n )  and  m)  is  said  to  be  well-posed  (Hadamard.  1923);  otherwise 
it  is  said  to  be  ill-posed.  Notice  that  iii)  may  depend  on  the  choice  of  the  metric  in  A  and 
in  Y. 

If  .4  is  linear,  continuous,  injective  and  R(A)  =  Y,  the  problem  of  solving  (2.1.1)  for 
r  is  trivially  well-posed:  indeed,  since  ,4  is  a  bijection  between  D(A)  and  Y .  existence  and 
uniqueness  of  the  solution  are  guaranteed.  Moreover  x  depends  continuously  on  y  because, 
when  R(A)  =  F.  A~l  is  continuous  (Riesz  and  Nagy,  1952). 

If  A  is  linear  but  not  injective,  the  solution  to  the  problem  of  (2.1.1)  is  no  longer  unique. 
Uniqueness  of  the  solution  can  be  easily  recovered,  for  instance,  by  introducing  the  concept 
of  normal  solution.  The  normal  solution  xn  to  (2.1.1)  is  the  solution  ortl  rgonal  to  the  null 
space  of  A,  N(A).  It  is  easy  to  see  that  x„  is  unique  and  that  it  can  be  characterized  as 
the  minimum  norm  solution.  If  .4  is  injective,  the  normal  solution  and  the  usual  solution 
coincide. 

If  we  relax  the  condition  7?(A)  =  Y  other  problems  arise.  The  solution  to  (2.1.1)  may 
no  longer  exist  since  y  may  not  belong  to  R(A).  For  example  the  data  y  may  be  affected  by 
an  error  dy  belonging  to  the  orthogonal  complement  to  the  range  of  .4.  /?(.4)x.  It  is  useful 
then,  to  introduce  the  concept  of  quasi-solution  (see.  for  example.  Tikhonov  and  Arsenin. 
1977 ).  Let  P  be  an  operator  that  projects  Y'  onto  R(A ),  then  7,  the  solution  to  the  equation 

.4;r  =  Py  (2.1.2) 

is  called  a  quasi- solution  of  (2.1.1 ).  It  is  obvious  that  7  exists  if  y  £  R{  A  )  •  ;•  Ri  .4  )  .  Notice 
that  if  //  £  /?( .4 ).  the  quasi -solution  and  the  solution  to  (2.1.1)  coincide. 

1  herefore  if  .4  is  linear,  continuous  and  /?( .4 )  is  closed,  the  problem  <4  finding  a  norma  1 
qnaM  solution  to  the  erpiation  (2.1.1)  is  well  posed,  since  the  normal  quasi  solution  always 


exists,  is  unique  and  depends  continuously  on  y.  (This  last  condition  follows  directly  from 
tie'  continuity  of  the  quasi-inverse  of  .4,  ,4+  being  defined  as  the  operator  that  maps 
1/  f  R(  A)  -j-  R(  A)1-  into  the  corresponding  normal  quasi  solution  of  (2.1.2).) 


In  many  practical  cases,  however,  R(A)  is  not  closed  (Kolmogorov  and  Fomine.  1 0-Sf J  • . 
So  even  the  quasi-solution  may  not  exist  and  if  it.  exists  can  be  unstable.  Consider,  foi 
example,  the  Fredholm  integral  equation  of  the  first  kind 
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J  K{t,s)x(s)ds  =  y(t)  c  <  t  <  d.  (2.1.3) 

a 

The  function 


.r( .s )  =  ,r(s)  +  N  smuts 


is  a  solution  to  (2.1.3)  with 


17(f)  =  y(t)  +  N 


.s)  s'm(u:s)ds. 


a 


In  the  usual  Li  metric  ||t/  —  t/||  — »  0  as  ut  — >  oo  (for  the  Riemann  Lebesgue  theorem!  while 
c  /  j;  ~  N.  So  with  a  suitable  choice  of  ,V  and  at  the  error  on  the  data  can  be  made 
arbitrarily  small,  while  the  distance  between  the  solution  can  be  arbitrarily  large. 
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2.2.  Ill-posed  problems  in  5?n 

Let  us  consider  the  system  of  equations 

Ax-  y  (2.2.1) 

where  .4  is  a  n  x  n  matrix  and  x  and  y  vectors  belonging  to  3?n.  The  problem  of  recovering 
x  given  .4  and  y  is  that  of  finding  the  inverse  matrix  A~l  of  .4.  If  the  determinant  of  .4 
is  equal  to  zero,  the  problem  has  no  solution  and  the  system  is  called  singular.  If  .4  is 
diagonalizable  and  some  eigenvalues  are  much  smaller  than  the  others,  the  system  is  said 
to  be  ill-conditioned  (Strang  1976),  since  small  errors  in  the  data  y  lead  to  unacceptable 
indeterminacy  in  the  components  of  the  solution  x.  In  such  cases  the  ratio  between  the 
largest  and  the  smallest  eigenvalue  of  A  is  taken  as  the  ill- conditioning  number ,  that  is  a 
measure  of  how  much  the  system  is  ill-conditioned.  Notice  that  whether  an  ill-conditioning 
number  leads  to  negligible  errors  or  not  depends  not  only  on  the  system  but  also  on  the 
accuracy  required. 

Even  in  the  case  of  huge  ill-conditioning  number,  however,  the  problem  of  solving  (2.2.1 ) 
is  not  ill-posed  in  a  classical  sense,  since  for  arbitrarily  small  errors  in  the  data,  the  solution 
is  arbitrarily  close  to  the  exact,  solution.  In  practice,  however, approximations  involved  in 
numerical  computations  lead  to  meaningless  solutions,  because  the  error  in  the  data  is  not 
arbitrarily  small. 

Let  us  consider  now,  more  closely,  the  problems  that  could  arise  in  numerical  compu¬ 
tations:  let  A,.  i  =  1 _ _  n  be  the  eigenvalues  of  .4.  It  is  easy  to  see  that 

•<',  =  7-;/.  1  =  T  •••,?' 

will  l.e  the  components  of  x,  the  solution  of  (2.2.1),  after  a.  suitable  transformation  ot 
1  <  m  a  di nates.  If  even  small  errors  affect,  the  entries  of  .4,  when  some  A,  are  sufficient  ly  close  to 


r> 

/rid.  rlir  corresponding  components  x,  of  the  solnt  ion  can  become  arbit  tarib  l.n  w. 

to  an  unhounded  solution.  As  a  matter  of  fact,  the  errors  arising  from  numerical  rompun  t 

approximations  could  be  sufficient;  therefore  even  numerical  problems  can  be  ill-post  d. 

3.  Shape  constraints  in  regularization 

Ill  posed  problems  can  be  successfully  turned  into  well  posed  pioblems  by  means  of  \  *  i  \ 
general  regularizing  techniques.  As  it  is  well  known  (Tikhonov  19-13.  1903).  these  techniques 
rely  on  the  assumption  of  some  smoothness  property  of  the  possible  solution.  Sometimes, 
however,  additional  and  useful  constraints  are  available;  for  example  the  solution  function 
may  be  necessarily  non-negative  or  a  monotonic  function  and  so  on.  In  this  chapter,  after 
discussing  the  role  of  compactness  in  regularization,  we  show  which  of  these  constraints  can 
be  embedded  in  the  classical  regularization  theory  and  how. 

.3.1.  Role  of  compactness  in  regularization 

1  Id  played  by  compactness  (see  Appendix  A  for  its  various  definitions  and  properties j  in 
the  solution  of  ill-posed  problems  was  clarified  by  Tikhonov  with  the  following  fundamental 
topological  Lemma  (Tikhonov  and  Arsenin,  1977): 

/a  o:  mu.  s’. 1.1  Suppose  that  the  operator  .4  maps  a  compact  set  F  C  A  onto  the  set  F  C  . 
A  anti  5  metric  spaces.  If  .4  :  F  — >  U  is  continuous  and  one  to-one.  then  the  inverse 
mapping  .4|f~'  is  also  continuous. 

My  means  of  this  Lemma,  if  the  solution  to  equation  (2.1.11  is  known  to  belong  to  a 
■  rip;  <•;  subset  of  A  .  say  }  .  and  if  the  pert  mix  ■>  1  data  is  kin  am  t  >  Ik -long  '<>  F .  I '  —  ■[ .  t 
)  .  .hr},  then  the  problem  of  finding  a  solution  to  (2.1.  t  is  trivially  well -posed  with 

o  ~j  ••  •  ’  to  /-  and  1  .  In  sin'll  a  ease  tin-  pioblem  is  .aid  to  in*  w  H~ post  d  m  tin  sensi  of 
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R<  mark:  The  compactness  recpiirement  is  a  strong  constraint  on  the  set  of  possible  solutions 
to  a  given  problem:  it  is  possible  to  produce  examples  in  which  wcll-poseclness  is  guaranteed 
without  any  compactness  requirement  (Groetsch,  1984). 

If  some  a  priori  constraints  on  the  shape  of  the  solution  are  known  and  if  these  con¬ 
straints  lead  to  the  definition  of  a  suitable  compact  set,  the  application  of  Lemma  3.1.1  is 
straightforward.  This  is  the  theme  of  the  next  sections. 

3.2.  The  selection  method 

A  useful  method  of  finding  an  approximate  solution  to  equation  (2.1.1)  is  the  selection 
method  (Tikhonov  and  Arsenin,  1977).  It  consists  in  calculating  the  operator  A  for  points 
belonging  to  a  given  sample  set,  looking  for  the  minimum  of  ||Ax  —  y||  in  a  suitable  norm. 
Such  a  method  is  powerful  from  a  computational  point  of  view  since  the  sample  set  can 
be  choosen  so  to  depend  only  on  a  finite  number  n  of  parameters  varying  in  finite  limits. 
Obviously  the  computed  solution  xn  and  the  exact  solution  xt  (if  xt  exists)  coincide  if  and 
only  if  X(  belongs  to  the  sample  set. 

Suppose  that  increasing  the  number  n  of  parameters  (and  therefore  the  dimension  of 
the  subspace  containing  the  sample  set)  ||.4.rn  —  y||  — >  0.  Let  us  assume,  therefore  that 
j|.4x„  —  y||  — ►  0  as  n  — »  oc.  It  is  easy  to  see  that  if  the  R(A)  is  not  closed  the  approximate 
solution  ||x„||  — ►  oo,  hence  xn  does  not  converge  to  xt.  In  order  to  guarantee  the  convergence 
of  to  .(•/.  compactness  of  the  sample  set  is  needed,  so  that  Lemma  3.1.1  applies.  If  the 
sample  set  is  not  compact  but  it  is  closed  and  bounded,  the  Lemma  3.1.1  is  still  valid, 
though  in  a  weaker  sense.  The  solution  xn.  in  fact,  is  only  weakly  convergent  1  to  the  true 
solution  j  (.  it  is  also  convergent  in  the  usual  sense  if  x,  lies  on  the  boundary  of  the  sample 
set  (  Berturo.  19S2). 


I  lie  •-I'iiitiiMi  .i„  is  -..till  t.i  ln>  weakly  convergent  to  rt  if  (j  „.  i/1  —  l  i'i.i/)  lor  n  — 
where  |  .)  is  a  suitable  <lol  product. 


-  x  v)/  t  A  . 


3.3.  Regularization  theory  and  shape  constraints 

When  no  compact  set  containing  the  possible  solution  of  (2.1.1)  can  lx*  found,  a  neve  ap 
proach  is  needed.  A  general  and  useful  approach  was  also  outlined  by  Tikhonov  <  1943.  1903 
and  is  called  standard  regularization  theory.  Let  us  briefly  summarize  the  main  point*'  of 
this  theory. 

The  fundamental  concept  of  the  theory  is  that  of  a  regularizing  operator.  Suppose  that 
the  equation  (2.1.1)  allows  x  =  x,  as  a  solution  when  ij  —  yt:  then  an  operator  Riy.r pi 
called  a  regularizing  operator  for  the  equation  (2.1.1)  in  a  neighborhood  of  x  —  . r ,  if: 

i)  3/>i  >  0  such  that  R(y,a)  is  defined  Vo  >  0  and  V;/  £  V  such  that  ||t/  —  t/(||  <  «S , : 

ii)  there  exists  a  function  a  =  a(6)  such  that  Vc  >  0  3A  <  th  such  that  V// 

II V  -  Vi\\  <  <5  =*  ||  JT  ~  J-/.II  <  f 

where  ,r(1  =  R(y,  n(h)) 

So  the  problem  of  finding  a  regularized  solution  to  an  ill -posed  problem  is  shifted  to 
that  of  finding  methods  to  construct  a  regularizing  operator.  Let  us  see  in  some  detail  on* 
of  these  methods. 

Construction  of  regularizing  operators  by  minimization  of  a.  smoothing  functumn ! 

It  is  possible  to  construct  a  regularizing  operator  for  ( 2.1.1 )  by  minimizing  the  foil,  .win 
lut.ctional  with  respect  to  x: 

'P"[.r,  y]  =  || Ar  -  ;/||  f  of >[./■]  .3.3.1 

[}  L  a  stabilizing  functional.  A  functional  if  defined  on  ()  '  Pi  A  •  . <  i  \  \\  ] i*  ;■  d. 

:n  Pi  A  i  is  .i  stabilizing  functional  for  the  equation  (2.1.1  )  if: 


9 


MAWWWA'MK’A'  IJW  VV  WWW  W  u »  k."U.'V»\tfV*v'  p  wiiwwwlwi 


i)  x(  belongs  to  the  domain  of  definition  of  Cl: 

ii)  Vt /  >  0,  {x  G  0  fi[x]  <  d}  is  a  compact  subset  of  0. 

Indeed  the  following  theorem  holds: 

Theorem  3.3.1  Let  A  denote  a  continuous  operator.  For  every  y  G  I  and  every  a  >  0.  there 
exists  a  .vn  G  O  for  which  the  functional  ^  attains  its  minimum. 

As  a  matter  of  fact  the  choice  of  Cl  can  determine  the  uniqueness  of  the  solution:  for 
example  if  D(A)  is  a  Hilbert  space  and  A  is  linear,  if  Cl  is  quadratic,  sufficient  condition 
for  the  uniqueness  of  the  regularized  solution  can  be  proved  (Tikhonov  and  Arsenin.  1977  s 
In  principle,  the  regularization  problem  is  completely  solved.  Sometimes,  however,  some 
additional  constraints  on  the  shape  of  the  solution  are  available.  Can  we  exploit  them  7 
Indeed,  the  following  Lemma  holds: 

Lrvnna  3.3.2  Let  A'  be  a  compact  topological  space.  Then  every  closed  subset  of  A  is 
compact. 

Theorem  3.3.1  is  based  on  the  compactness  of  the  subsets  where  Cl  is  bounded  and 
therefore  is  still  true  even  if  the  set  of  possible  solutions  is  a  closed  subset  of  D(  .4 ).  Therefore, 
if  the  additional  constraints  lead  to  the  definition  of  some  closed  subset  of  D(A).  they  can 
In  easily  exploited  in  the  framework  of  regularization  theory. 

ft  <  mink:  these  sets  do  not  need  to  be  compact.  The  regularizing  scheme  itself  provides 
compact  ness  of  the  set  in  which  the  solution  is  actually  searched:  if  the  constraints  define  a 
compact  set.  the  Lemma  3.1.1  is  sufficient  to  guarantee  well-posedness  of  the  problem. 

3.  I.  Compact  subsets  of  functional  spaces 

I-:  om  t  he  preceding  sections,  it  turns  out  that  given  an  ill-posed  problem  and  some  a  jirmri 
constraints,  it  is  important  to  determine  whether  such  constraints  define  a  compact  subset 
ot  at  least  a  closed  subset  of  a  suitable  functional  spare.  Let  us  examine  souk-  examples  oj 

el  <  l  •  ( it  /.  >  and  C" . 


T hr  set  of  bounded  non-decreasing  (non-inm  vising ■  func'io!  -  >  ;• 

Tlji-  ;>roof  (see  Taylor.  1965.  for  example)  relies  upon  the  iaet  th.  t  i li« ■  mimo,  :  <>f  .i: 
unity  points  of  a  inonotonie  bounded  function  is  a t  most  enumei able. 

The  set  of  convex  functions  is  compact.  This  result  follow-,  tnvially  from  the  compa<  t 
tie--  of  tlu*  set  above,  since  each  convex  function  is  the  integral  oj  a  -uitahie  non-dee; <  a -in . 
function. 

The  set  of  bounded  piece-wise  constant  functions  is  neither  closed  nor  compact.  1'  : 
not  compact  since  it  is  everywhere  dense*  in  L>  (which  is  trivially  not  compact),  h 
clost'd  since  any  continuous  function  is  an  accumulation  point  of  this  set. 

It  is  not  easy  to  find  compact  subsets  of  C'i] .  The  set  of  bounded  non  in  -gat  ive  funct  i<  an. 
for  extuiqile,  is  not  compact.  Consider  in  C[0.  1] 

T  -  {-r  l-r(t)!  <  1  .  1  £  jO.lJ}. 

T  is  closed  and  bounded  (obvious),  but  not  compact.  Indeed,  let  S  ■—  { .r , } ,  e  \  be  a  sequent 
of  functions  with  x„(t)  -  i" .  Any  subsequence  of  S  cannot  converge  in  T.  since  in  ("[(),  1] 
'lie  convergence  is  uniform  convergence  while  t”  — >  0,  if  0  <  /  <  1.  and  t"  —  1.  if  /  --  1.  So 
r  i-  not  compact. 

R<  mark:  This  counterexample  shows  that  in  C°  even  the  sets  of  monotonic  and  convex 
fuue'ioiis  are  not  compact. 

A>  a  conclusion,  the  constraints  of  monotonicity  and  of  convexity,  delining  compact 
-u!  >~et s .  can  be  useful  in  regularization  either  via  the  selection  method  or  \  i:i  standard  tech 
us  jii< 's  ( since  any  compact,  set  is  closed,  see  Appendix  A  for  detail ).  The  posit  ivity  const  mint 
ran  Is  used  only  as  a  shape  constraint  in  classical  r<  gularization  theoiv  and  in  a  weaket 
•  (  'I  |S<  •  Ill  the  selection  method,  while  piece- wise  constant  functions,  though  representing  sig 
uiiicative  a  prion  knowledge  on  the  shape  of  the  solution,  cannot  he  embedded  in  <’itlin  .  >< 


4.  Connection  with  mathematical  programming  (MP) 


M<)'t  of  the  problems  fared  in  the  framework  of  Hilbert  spaces  are  in  fact  usually  eifhei 
intiinsically  discrete  problems  or  problems  allowing  only  numerical  solution.  In  (his  chaptei 
the  cases  of  the  selection  method  and  of  the  regularization  with  shape  constraint,  discussed 
in  the  preceding  sections,  an'  analyzed  in  this  respect  as  examples  of  mathematical  p:  ■ 
gi  a  mining  problems. 

4.1.  Selection  method  as  a  MP  problem 

As  we  have  seen  in  the  previous  chapter,  if  the  condition  of  section  3.1  applies,  an  ay 
pioximate  solution  to  the  equation  (2.1.1)  can  be  found  by  means  of  the  selection  method. 
In  practice  the  problem  has  to  be  solved  numerically:  consider  for  example  the  Fredholm 
integral  equation  of  the  first  kind 

b 

J  K(t,  .s).r(.s)d.s  =  y{t)  c  <  t  <  d  (4.1.1' 

a 

where  ,r(s )  belongs  to  a  set  F  of  decreasing  uniformly  bounded  functions.  F  is  compact  (  see 
section  3.4),  therefore  if  y(t)  £  U  =  AF  the  problem  is  well-posed  in  the  sense  of  Tikhonov. 
In  order  to  find  an  explicit  solution  we  can  replace  the  integral  with  a  sum  over  a  grid  with 
a  nodes.  Let  ,r,  (;  =  l....,/j)  be  the  value  of  the  unknown  vector  x  at  the  node  i  and 

!j ,  ' =  1 . m)  the  components  of  the  data  vector  y.  The  problem  is  to  find  a  bounded 

vector  minimizing  the  functional 

m  n 

'Hx-yj  =  ~  Ml 

;=i  «=i 

uudei  the  constraint  that  the  components  of  x  are  decreasing.  It  is  easy  to  '-how  that  tin- 
1  i»n  iiaint  can  be  expressed  as  a  positivity  constraint  on  the  values  of  the  derivative  ot 


tin-  function  at  each  node.  In  the  discrete  case  t iris  reduces  to  the  fact  that  suitable  linear 
combinations  of  the  neighbor  nodes  have  to  be  greater  than  zero.  For  example  in  the  nemo! 
neighbor  approximation  we  have 


(•F+l  —  l)  ,  ,  ,  -i  .-) 

2 

In  these  terms  the  problem  is  now  a  typical  problem  of  quadratic  programming  (see 
Appendix  B  for  main  definitions  and  results  of  mathematical  programming  problems).  In¬ 
deed  in  the  general  case  the  only  problem  concerns  the  explicit  form  of  the  constraints.  It 
must  be  possible  to  write  them  as  follows  (see  Appendix  D): 


j/,(x)  <  0  i  =  1.  (4.1.3) 

wliere  ,/t  are  scalar  functions  (they  need  to  be  linear  or  at  most  quadratic  to  define  a 
qr.ndtatic  programming  problem).  Notice  that  (4.1.2)  can  be  immediately  rewrit  ten  like 
.  .  -1  Butman  and  Cabral  (1974)  have  shown  that  performing  a  suitable  transformation. 

■  ■  nist  taint  of  monotonicity,  convexity,  uniinodality  and  selective  non- negativity  can  till 
-i  "bitten  m  the  form  (4.1.2).  In  this  case,  as  shown  before,  only  the  monotonicity  and  the 
ci>:i\v\ity  constraint  can  be  properly  used.  As  we  will  see  in  the  next  section,  however,  till 
:  :n  are  shape  constraints  that  can  be  useful  in  regularization. 

1.2.  Regularization  with  shape  constraint  as  an  MP  problem 

I.'  '  u-  illustr  .is  section  by  means  of  the  same  example  of  the  previous  one.  Again  the 
;  ••  ■■■!'  is  to  solve  the  Fredholm  integral  equation  of  the  first  kind  (4.1.1).  This  fim*'  since 
1  ;■  i  fie  -* -t  /•  is  not  compact  or  ij  does  not  belong  to  L  —  AF  .  st;in<lard  reguhtri/.iition 
•'•i-hiiiqui's  of  the  kind  described  in  section  3.3  are  needed.  Suppose  moreover  that  some 


lut’hej  information  is  available  and  that  they  eorrespond  to  constraints  on  the  soluiton 
- titling  closed  subsets  of  the  domain  of  the  opet  a  tor.  If  t  hes<  const  raints  can  be  writ  ten  m 


the  form  (4.1.2)  the  problem  of  minimizing  the  discrete  functional  corresponding  to  (3.3.1  ) 
subject  to  such  constraints  is  again  a  typical  stable  problem  of  quadratic  programming. 


Remark:  A  generic  mathematical  programming  problem,  even  if  quadratic  or  linear,  is  not 
necessarily  stable.  As  a  matter  of  fact  the  well-posedness  relies  on  the  strong  assumption 
that  the  functional  to  minimize  is  a  stabilizing  functional.  If  this  is  not  the  case,  the  problem 
ha,-  to  be  regularized  following  standard  techniques  (Tikhonov  and  Arsenin.  1977). 

Remark:  While  any  regularized  problem  of  the  type  described  in  section  3.3  gives  rise 
to  a  well-posed  mathematical  problem,  the  application  of  Kuhn-Tucker  theory  and  of  the 
gradient  method  are  subject  essentially  to  the  fulfillment  of  some  convexity  properties  of 
the  functions  involved  (see  Appendix  B)  and  therefore  they  are  guaranteed  only  in  the  case 
of  linear  operators  and  a  quadratic  stabilizing  functional. 


5.  Conclusion 


lu  this  note  we  analysed  the  role  played  by  shape  constraints  in  ill-posed  problems.  The 
key  concept  has  been  that  of  compact  set.  If  the  shape  constraints  lead  to  the  definition  of 
a  compact  set,  regularization  is  straightforward.  Indeed  the  shape  constraint  itself  provides 
sufficient  conditions  for  the  continuity  of  the  dependence  of  the  solution  on  the  data.  If 
the  shape  constraints  define  at  least  a  closed  set,  then  they  can  be  an  useful  addition  to 
standard  regularization  approaches.  While  a  suitable  functional  provides  stability  on  the 
data,  shape  constraints  allow  to  recover  a  solution  closer  to  the  correct  one,  by  taking  into 
account  significative  additional  a  priori  knowledge  on  the  shape  of  the  solution. 

In  both  cases  constraints  that  do  not  define  at  least  a  closed  set  cannot  be  embedded 
m  the  regularizing  step.  In  particular  this  implies  that  the  a  prion  knowledge  concerning 
piece- wise  constant  or  piece-wise  continuous  functions,  though  in  principle  significant  for 
many  early  vision  problems  (the  reconstruction  of  the  3D  structure  of  a  scene  and  the 


recovery  of  the  albedo  for  example)  cannot  be  used  within  any  classical  regularizing  schema. 
This  is  an  additional  argument  that  motivates  the  use  of  Marian  Random  Fields  models  fm 
exploiting  a  prion  information  about  discontinuities  and  their  properties  (see  Marmquiu 
et  al.  1985).  A  different  regularizing  approach  that  can  exploit  constraints  of  this  type, 
considering  discrete  and  quantized  formulations,  will  he  discussed  in  a  forthcoming  papei 
i  l’i iggio  and  Verri). 

Finally,  the  discrete  problem  that  has  to  be  faced  solving  an  ill-posed  problem  lias 
been  analysed  as  a  mathematical  programming  problem:  in  the  interesting  case  of  lineai 
operators  it  becomes  a  standard  stable  problem  of  quadratic  programming.  In  particular, 
all  the  results  of  convex  programming  regarding  local  and  global  convergence  of  the  gradient 
method  algorithm  are  guaranteed  to  apply. 
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Appendix  A:  Compact  sets  in  topological  and  metric  spaces 

As  we  have  seen  before  the  concept  of  compact  set  is  fundamental  in  the  regulariza¬ 
tion  of  ill-posed  problems.  Unfortunately,  there  exist  different  definitions  of  compact  set. 
Disregarding  historical  problems,  here  is  a  summary  of  the  main  definitions  and  properties 
concerning  compact  sets  that  we  adopted  in  this  note. 

Let  X  be  a  topological  space.  An  open  covering  of  S  C  A  is  a  family  F  of  open  sets  ::. 
.V  such  that  S  C  (J-er  i. 

S  C  X  is  compact  if.  for  every  open  covering  T  of  S .  there  exists  a  finite  subfamily  of 
F  that  also  covers  S. 

Remark:  A  closed  set  is  not  necessarily  compact  (consider  the  real  line).  A  compact  set  is 
not  necessarily  closed.  Compact  sets  are  always  closed  in  Hausdorff  spaces  (a  topological 
space  is  a  Hausdorff  space  if  for  each  pair  of  distinct  points  xt  and  xj .  there  exist  two 
disjoint  neighborhoods  containing  them). 

In  topological  spaces  the  following  Lemma  holds: 

Li  hi m a  A.l  If  T  C  X  is  compact,  then  for  every  infinite  5  C  T .  S'  H  T  ^  0.  (S'  is  the  set 
of  accumulation  points  of  S). 

Notice  that  the  converse  of  Lemma  ,4.1  is  not  true  in  general.  Now  let  A  be  a  metric 
space  (and  henceforth  a  topological  Hausdorff  space)  then  we  have: 

Lemma  A. 2  If  T  C  A'  and  for  every  infinite  S  C  7\  S'  d  T  ^  0,  then  T  is  compact. 

R<  mark:  Combining  Lemma  A.l  and  .4.2  the  usual  definition  of  compactness  in  metric 
■'paces  can  be  obtained:  a  set  S  Q  X,  X  a  metric  space,  is  compact  if  for  every  sequence  of 
points  in  5  there  is  a  subsequence  converging  to  a  point  of  5. 

Furthermore,  in  metric  spaces  the  concept  of  bound nine .«.■<  can  be  defined,  so  that  the 
iollowing  Lemma  can  be  proved: 

L<  mma  A.H  If  S  C  A'  is  compact  then  S  is  closed  and  bounded. 


hr  converse  of  Lemma  .4.3  is  not  true  in  general  (  see  section  3  1  lot  a  count.  ic.\ain|  >.■ 


a.  ink:  In  ft"  the  converse  of  Lemma  .4.3  holds.  Indeed  in  I?",  tor  the  Bond  theorem.  am 
bounded  set  has  an  accumulation  point:  so,  if  it  is  closed,  it  is  also  compact. 

[t  follows  that  discretization  makes  a  problem  well  posed  ( bit*  ill-conditioned  possibly  i. 


Appendix  B:  Mathematical  programming:  Definitions  and  main  results 


In  Vision,  when  a  regularized  problem  has  to  be  solved,  numerical  methods,  based  on 
di  -in  et izing  the  original  continuous  formulation,  arc  usually  needed.  1  liese  numerical  met  h 
>  h always  lead  to  classical  problems  of  mathemati.  al  programming.  As  may  be  expected  m 
'  h.  M-  cases  Tikhonov  regularization  theory  and  mathemat ic;d  programming  theorems  pre 
. i let  the  same  results  in  terms  of  existence  and  uni.pieness  of  the  solution  (see  sections  hi 
end  -1.2).  Here  we  review,  for  the  sake  of  completeness,  the  main  definitions  and  resuits  < .1 
mathematical  programming  theory  (for  more  details  set-  Anew  et  al..  1038.  fot  example!. 

her  us  consider  the  problem  of  finding  a  minimum  for  a  given  functional  ~  =-  ;i  z  on 

a  set  G  —  {z  /  gt(z)  <  0  /  =  1 . ?»}  where  z  -  ( . z„  )  £  L  C.  R"  and  </,  are  scalar 

lift  ions.  If  the  functions  ^  .and  g,  ( i  —  1 . m)  are  linear,  the  problem  is  called  a  lunar 

ii:  m;  n:  mm  ni.i]  problem,  otherwise  v  on-linear.  In  both  cases  it  is  a  mathematical  programming 

; i :  a  hi i  m. 

Typically  the  problem  of  finding  conditional  extrema  of  a  given  functional  is  -oiv.fi 
b\  means  of  the  Lagrange  multipliers  theory.  Classical  theorems  on  Lagrange  multipliers 

;fi. ■  only  necessary  conditions  for  the  existence  of  such  mill tiphers:  Kuhn-  1  uck.-r  theory, 
in  tin  n.  fills  the  gap.  providing  sufficient  conditions  tor  then  e.X’stence  .  obviously  close!;, 
lelat.d  to  the  existence  of  extrema  of  functionals).  Ibis  tlieoiy.  therefore.  is  useful  in 
ino-t  of  the  mathematical  programming  problems,  le  t  us  review  briefly  the  main  result'  o| 
l\  ■  , :  ; !  I  I  uck<  i  theoi  y. 

l\  nil  ii  ■  [inker  theory 


*  s’ 


tmy* 
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Let  us  call  the  conditional  problem  stated  above  P.  1  and  associate  with  it  the  f< 


mowing 


Lagrangian  form 


<I>(z,  w)  =  v?(z)  +  ^2  u'i9>(z) 


where  w  :  (u'i . wm )  with  if,  €  3£+,  i  =  1  It  is  easy  to  see  that  if  the  pair  (z'.w'i 

i-  a  saddle  point  for  the  above  Lagrangian  form  z'  is  a  solution  of  PA.  Let  us  call  P.2  the 


problem 


of  finding  a  saddle  point  for  the  Lagrangian  form  thus  the  following  Lemma 


Lemma  B.l  Given  PA  and  P.2,  if  the  pair  (z',w')  is  a  solution  to  P.2  then  z'  is  a  solution 
to  PA. 

To  prove  the  converse  of  Lemma  BA,  i.e.  to  show  the  equivalence  between  PA  and  P.2, 
some  constraints  on  the  functions  and  g, ,  i  =  1  ,...,m  are  needed:  more  precisely: 

Thuntm  B.2  ( Kuhn -Tucker)  Let  yp(z)  and  g,{z),  i  =  1  ,...,m  be  convex  on  Z  =  {z  /  z,  > 

I),  i  -  1 . ri}.  If  there  exists  z°  €  Z  such  that  g,(z°)  <  0,  i  =  l,..., in.  then  z'  is  a  solution 

to  P.  1  if  and  only  if  3  w'  such  that  the  pair  (z',  w')  is  a  solution  to  P.2. 

In  the  case  of  C1  functions  the  celebrated  Kuhn- Tucker  conditions  can  be  introduced. 
They  guarantee  necessary  conditions  for  the  existence  of  a  solution  to  a  saddle  point  problem. 
Under  convexity  assumptions  the  Kuhn-Tucker  conditions  become  sufficient,  henceforth 
guaranteeing  the  existence  of  a  solution  to  the  associated  mathemat’cal  problem.  (If  r_ 
is  strietlv  convex  it  also  turns  out  that  the  solution  is  unique).  In  obvious  notation  they 


7=l  J 


■X —  <  o,  j  =  1 . rn 

aw} 


WWW 


jf 


(i  radiant  method 


Let  us  now  review  briefly  the  gradient  method,  which  is  one  of  the  most  useful  methods 


r  finding  saddle  points  of  a  given  function.  It  consists  essentially  in  finding  flic*  solution  of 


the  following  system  S  of  differential  equations 


dz, 

—7-  =  0  if  — —  >  0  and  z,  -  0 

at  dz, 


dz _  d$_ 

dt  dz. 


otherwise;  i  -  1 . n 


dw,  d$ 

— —  =0  if  — —  <  0  and  ir,  =  0 

dt  dir  j 


dw  j 


dt  dir 


ot  h'Twise;  /  -  1 . in 


is  ;i  parani'*ter.  Now  if  the  pair  fz'.w')  is  a  saddle  point  for  •I’lz.  wi  it  follows  that: 


d<P 

(z'.  w' )  _■  0 
dz, 


i  ; . n 


J 


hi  particular  if  §J(z',w')  >  0  then  z\  =  0  and  if  |Jh(z',  w')  <  0  then  «•'  =  0.  Without 

1""'  generality  (just  for  notational  convenience)  suppose  in  the  sequel  that  for  /  =  1 p. 

■I'  -  ">  w')  =  0-  while  for  i  =  p+  1, n  f£(z\  w')  >  0  and  that  for  j  =  1 . f/. 

■  V  '")  ^-(z',  w')  =  0.  while  for  j  =  q  +  1 . m  j^-(z',  w' )  <  0. 

The  following  theorem,  now,  guarantees  local  convergence  of  the  gradient  method. 

Theorem  B.3  Let  <3>(z,w)  have  a  saddle  point  (z',w')  under  the  constraint  z  £  Z  where 

Z  ~  { z  /  z,  >  0,i  =  and  w  6  W  where  W  —  {w  /  u>;  >  O.y  =  1 . u,}  and 

let  <I>  he  analytic  in  some  neighborhood  of  (z',w').  Suppose  further  that  the  matrix  of  the 
M-rond  derivative  of  $  in  the  first  p  components  of  z  defines  a  positively  defined  form  and 
that  >  0,  i  =  1,  ...,p  and  Wj  >  0,  j  =  1, q.  Then  for  any  pair  (z".  w"  )  in  a  sufficiently 
sun 1 11  neighborhood  of  (z\  w'): 

there  is  a  unique  solution  z  =  z(f.z",  w"  )  and  w  =  w(t,z",  w")  to  the  system  .9  such 

that: 

ri)  lim,.^  z(f,z",  w")  =  z'  and 

m)  in  any  limit  point  w°  of  the  function  w  =  (t,  z",  w")  as  1  ->  oc.  the  pair  (zh  w° )  is 
saddle  point  of  <l>(z,w). 

R(  mart,-;  1  he  classical  theorems  of  existence  and  uniqueness  of  the  solution  for  differential 
-W't<  m  of  equation  cannot  be  used,  since  no  assumption  is  actually  made  on  the  continuity 
d  the  derivatives  of  the  variables. 

Before  stating  the  theorem  on  global  stability  of  the  gradient  method  the  following 
definition  is  needed: 

d  W  '  1 . a  and  Wj\t)  ;  =  solution  of  the  system  S’  are  ;i  regular  solution  if 

'  ■ 11  11  d  L/ )  —  b.  i  1 . ri  and  aq(  t j, )  —  0,  j  =  1 , .. .,  m  wit h  (E  A  tor  some  sequence 


,  '  nrli  l  hat  iv  >  0  and  t„  —  0,  tli<  re  is  some  /  •  0  Midi  t  !i;ii  0.  /  1 . n 

'---i  0.  /  -  1 . in  for  0  <  t  <  t. 

i  m  D-4  Lot  d>(z,w)  bo  a  strictly  convex,  continuous  and  twice  differentiable  funcMoi, 
in  /  c  Z  and  w  G  TV.  Let  the  system  S  have  a  regular  solution  with  respect  to  any 
;  >a  u  •■■/:'.  w" )  where  z"  G  Z  and  w"  G  TV.  Then  there  is  a  unique  tegular  solution  of  t ! 

.,n  S  with  any  initial  position.  Ftirthermore  if  <I>  has  a  saddle  point  in  (  z' .  w' )  under  t In 
eou-r  raints  z  G  Z  and  w  G  IV.  z'  is  uniquely  determined  and  any  solution  of  S  converge- 


/,’■  i;  ■  /.•;  Actually  by  introducing  suitable  strictly  increasing  functions  />,.  j  -  1 . m  ofam 

die  -ueh  that  pf(  0)  --  0.  j  —  1 . ///.  the  condition  of  strict  <-onvexity  in  theorem  /j.  i 

be  relaxed  to  convexity  if  one  applies  the  gradient  method  to  the  ■modified  La.giangian 


$,,(2.  w)  =  ( z  )  f  ^  »',/'J[<//(z>]. 

j  -- 1 

I:.  conclusion  Theorem  B.4  guarantees  global  convergence  of  the  gradient  method  for 
•  ;  programming  (including  therefore  the  important  case  of  quadratic  programming!; 
modified  Lagrangian  form  above  allows  the  successful  extension  of  the  gradient  method 
,<■  broad  class  of  linear  programming  problems. 
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